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(57) It is required to protect the copyrights and the 
like of partial images which form respective parts of an 
image obtained by reading an image, exchanged using 
a print as a medium, by an image scanner or the like. 
Input image data is divided into a plurality of image re- 
gions having different features, digital watermarks, 
which are embedded in the detected image regions by 
embedding methods corresponding to the features of 
the image regions, are extracted, and the availability of 
the input image is checked on the basis of the extracted 
digital watermarks. 
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Description 

FIELD OF THE INVENTION 

[0001] The present invention relates to image 

processing, and, more particularly, to embedding and 
exlracLion of digital walemnark infcrmalion in an image 
on which images having different features are mixed, 
and an image process of a digital document or the like. 

BACKGROUND OF THE INVENTION 

[0002] In recent years, computers and networks have 
advanced remarkably, and many kinds of information 
such as text data, image data, audio data, and the like 
are handled on the computers and networks. Since 
these data are digitized, it is easy to fomn copies of data 
with equivalent quality. For this reason, in order to pro- 
tect the copyrights of data, copyright infomiation. user 
information, and the like are often embedded as digital 
watermark information (to be simply referred to as "dig- 
ital watermark" hereinafter) in image data and audio da- 
ta. 

[0003] "Digital watermarking" is a technique for em- 
bedding another infomriation, which is visually and au- 
dibly imperceptible to a human being, in secrecy in im- 
age and audio data by a predetermined process of these 
data. By extracting a digital watcmnark from image and 
audio data, the copyright information, user infomnation, 
identification infonnation, and the like of that data can 
be obtained. With such information, for example, per- 
sons who made illicit copies, and apparatuses used to 
form illicit copies can be traced from the illicitly copied 
digital data. In other words, digital watermarking can be 
applied to protection of the copyrights and the like of 
images, anti-counterfeit technology, various kinds of in- 
formation recording, and the like. 
[0004] Conditions required of such digital watermark- 
ing are as follows. 

(1) Quality: Digital watermark information must be 
embedded to be imperceptible, i.e., with least qual- 
ity deterioration of source digital infonnation. 

(2) Robustness: Information embedded in digital in- 
fonnation must remain undisturbed, i.e., embedded 
digital watermark information must never be lost by 
editing or attacks such as data compression, a filter 
process, and the like. 

(3) Information size: The information size of infor- 
mation that can be embedded must be able to be 
selected in accordance with different purposes of 
use. 

[0005] These conditions required for digital water- 
marking normally have a trade-off relationship. For ex- 
ample, upon implementing robust digital watermarking, 
relatively large quality deterioration occurs, and the in- 
formation size of information that can be embedded be- 



comes smalt. 

[0006] Taking a multi-valued still image as an exam- 
ple, digital watermarking can be roughly classified into 
two methods, e.g., a method of embedding in the spatial 
5 domain and a method of embedding in the frequency 
domain. 

[0007] Examples of the method of embedding a digital 
watermark in the spatial domain include an IBM scheme 
(W. Bender D. Gruhl, & N. Morimoto, "Techniques for 
10 Data Hiding", Proceedings of the SPIE, San Jose CA, 
USA, February 1995), G.B. Rhoads & W. Linn, "Stcga- 
nography method employing embedded calibration da- 
la", US Patent No. 5,636,292, and the like, which employ 
patchwork. 

15 [0008] Examples of the method of embedding a digital 
watennark in the frequency domain include an NTT 
scheme (Nakamura, Ogawa, & Takashima, "A Method 
of Watermarking in Frequency Domain for Protecting 
Copyright of Digital Image", SCIS' 97-26A, January 
1997). which exploits discrete cosine transformation, a 
scheme of National Defense Academy of Japan (Onishi, 
Oka, & Matsui, "A Watermarking Scheme for Image Da- 
la by PN Sequence", SCIS' 97-26B, January 1997) 
which exploits discrete* Fourier transformation, and a 
scheme of Mitsubishi and Kyushu University (Ishizuka, 
Sakai, & Sakurai, "Experimental Evaluation of Stega- 
nography Using Wavelet Transfonn", SCIS' 97-26D, 
January 1997) and a Matsushita scheme (Inoue, Miya- 
zaki, Yamamoto, & Katsura, "A Digital Watermark Tech- 
nique based on the Wavelet Transform and its Robust- 
ness against Image Compression and Transformation", 
SCIS' 98-3.2.A, January 1998) last two of which exploit 
discrete wavelet transformation, and the like. 
[0009] Also, as methods to be applied to a binary Im- 
age such as a digital document fonned by text, line fig- 
ures, and the like, a method of manipulating spaces in 
a text part (Nikkei Electronics, March 10,1 997 (no. 684, 
pp. 1 64 - 1 68). a method of forming a binary image using 
binary cells (density patterns) each consisting of 2 x 2 
pixels {Bit September 1 999A/ol. 31 , No. 9), and the like 
are known. 

[001 0] These methods are designed as pairs of digital 
watermark embedding and extraction processes, and 

are basically incompatible to each other. In general, 
methods of embedding a digital watermark in the spatial 
domain suffer less quality deterioration, but have low ro- 
bustness. On the other hand, methods that exploit the 
frequency transformation suffer relatively large quality 
deterioration but can assure high robustness. That is, 
these methods have different features, i.e., some meth- 
ods can assure high robustness but have a small infor- 
mation size of information that can be embedded, others 
can assure high quality but have low robustness, and 
so forth. Also, the embedding methods used for multi- 
valued images cannot be applied to binary images in 
principle. 

[0011] Color images, monochrome text images, line 
figures, and the like are often observed when they are 



25 



30 



35 



40 



45 



50 



2 



BNSDOCID: <EP 1349370A2 I > 



3 



EP 1 349 370 A2 



4 



displayed on a monitor display, and are often printed. 
Recently, prints with very high image quality can be cre- 
ated using not only a color copying machine but also an 
inexpensive printer such as an ink-jet printer or the like. 
In addition, since an expensive color image scanner, 
and convenient image processing software and image 
edit software which run on a personal computer have 
prevailed, an image can be scanned from a print with 
high image quality, and an image having equivalent 
quality in practical use can be reproduced. Furthermore, 
an image is scanned from a print with high image quality, 
and monochrome binary character images, line figures, 
and the like can be extracted and diverted. 
[0012] Various digital watermarking methods are 
available in correspondence with features, especially, 
those of image data in which a digital watermark is to 
be embedded, and a suited embedding method differs 
depending on image data. In each of these plurality of 
methods, digital watermark embedding and extraction 
processes are paired, but the different methods are in- 
compatible to each other. For this reason, an embedding 
method dedicated to a multi-valued image is used for 
multi-valued images, and that dedicated to a binary im- 
age is used for document images. Nowadays, however, 
digital images shown in Figs. 5 and 6, on which a photo 
image taken by a digital camera, a document created 
using wordprocessing software, and the like are mixed, 
are often created and printed. A digital watermarking 
method which can effectively applied to such digital im- 
age on which a plurality of image regions with different 
features are mixed and its print has been demanded. In 
the following description, a plurality of images having 
different features will be referred to as "dissimilar imag- 
es", and an image on which dissimilar images are mixed 
will be referred lo as a "mixed image". 
[0013] Also, it is demanded to protect the copyrights 
and the like of individual dissimilar images which form 
a digital image which is scanned from a print of a mixed 
image using an image input apparatus such as an image 
scanner or the like. In other words, it is demanded to 
protect the copyrights and the like of individual images 
(to be referred to as "partial images" hereinafter) which 
form respective parts of Images obtained by scanning 
an image, which is exchanged using a print as a medi- 
um, by an image scanner or the like. 
[0014] In recent years, as for security measures for 
office documents, the idea based on ISO 1 5408 has glo- 
bally spread, and such technical field is becoming in- 
creasingly important in that respect. As one of security 
management methods of document information, vari- 
ous kinds of digital watermarking techniques mentioned 
above have been proposed and used. 
[001 5] Security management can be used for various 
purposes such as illicit copy prevention of data, preven- 
tion of leakage or tampering of important information, 
copyright protection of document information, billing for 
use of image data and the like, and so forth, and various 
digital watermarking techniques have been proposed in 



correspondence with those purposes. For example, as 
a technique for imperceptibly embedding watermark In- 
formation in digital image data, a method of computing 
the wavelet transfonns of image data and embedding 

5 watemnark information by exploiting redundancy in the 
frequency domain (disclosed in Japanese Patent Appli- 
cation No. 10-278629), or the like is known. 
[0016] On the other hand, a binary image such as a 
document image has less redundancy, and it is difficult 

10 to implement digital watermarking for such image. How- 
ever some digital watermarking methods (to bo referred 
to as "document watermarking" hereinafter) that utilize 
unique features of document images are known. For ex- 
ample, a method of shifting the baseline of a line (Jap- 

15 anese Patent No. 3,136,061), a method of manipulating 
an inter-word space length (U S. Patent No. 6,086,706, 
Japanese Patent Laid-Open No 9-186603), a method 
of manipulating an inter-character space length (King 
Mongkut University, "Electronic document data hiding 

20 technique using inter-character space", The 1998 IEEE 
Asia-Pacific Conf. On Circuits and Systems, 1998, pp. 
419 - 422, a method of handling a document image as 
a bitmap image expressed by two, black and white val- 
ues (Japanese Patent Laid-Open No. 11-234502), and 

25 the like are known. 

[0017] The above methods are characterized in that 
the user cannot perceive watermark information embed- 
ded in an image (to be referred to as "invisible water- 
marking" hereinafter). Conversely, a method of embed- 

30 ding watemnark information which clearly indicating that 
the watermark information is embedded (to be referred 
to as "visible watermarking" hereinafter) is also pro- 
posed. For example, Japanese Patent Application No. 
1 0-352619 discloses a method of embedding a revers- 

35 ible operation result of an original image and an embed- 
ding sequence by comparing the pixel position of the 
original image and the shape of a watermark image to 
be embedded so that the watermark information is vis- 
ible to the user. 

40 [001 8] Digital watermarking basically aims at embed- 
ding of some additional information in image data itself, 
and protects an original image using the embedded ad- 
ditional information (e.g., prevention of unauthorized 
use, copyright protection, protection of tampering of da- 

45 ta, and the like). In other words, digital watermarking 
does not assume any purposes for inhibiting the user 
from viewing an original image itself or for allowing only 
a user who has predetemriined authority to copy data. 
[0019] Protection of an original image is applied to the 

50 entire image. For this reason, the user cannot often view 
or copy even an image contained in a protected image, 
which need not be protected. 

SUMMARY OF THE INVENTION 

55 

[0020] The present invention has been made to ad- 
dress at least one of the aforementioned problems, and 
has as its object to facilitate the embedding of a digital 
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watermark in an image on wliich image regions having 
different features are mixed. 

[0021] The present invention provides an image 
processing apparatus comprising; a detector, arranged 
to divide an input image into a plurality of Image regions 
having different features; an embedding section, ar- 
ranged to embed digital watermarks in the respective 
detected image regions by embedding methods accord- 
ing to the features of the image regions; and an integra- 
tor, arranged to integrate the image regions embedded 
with the digital watermarks into one image. 
[0022] An embodiment is operable to extract a digital 
watermark from an image on which image regions hav- 
ing different features are mixed, and check the availa- 
bility of the image. 

[0023] The present invention provides an image 
processing apparatus comprising: a detector, arranged 
to divide an input image into a plurality of image regions 
having different features; an extractor arranged to ex- 
tract digital watermarks embedded in the respective de- 
tected image regions by embedding methods according 
to the features of the image region; and a determiner, 
arranged to determine availability of the input image on 
the basis of the extracted digital watermarks. 
[0024] An embodiment is operable to check the 
availability of an image for each image region. 
[0025] The present invention provides an image 
processing apparatus comprising: a detector, arranged 
to divide an input image into a plurality of image regions 
having different features; an extractor, arranged to ex- 
tract digital watermarks embedded in the respective de- 
tected image regions by embedding methods according 
to the features of the Image region; and a determiner, 
arranged to detemnine availability of the input image on 
the basis of the extracted digital watemriarks. In an em- 
bodiment, the determiner determines availability of an 
image process for each of the detected image regions. 
[0026] An embodiment is operable 

to control a process for image regions of image 
information. 

[0027] The present invention provides an image 
processing apparatus comprising: an input section, ar- 
ranged to input digital image information; a detector, ar- 
ranged to recognize a predetermined image region in- 
cluded in the input image information; a generator, ar- 
ranged to generate authentication information required 
to control a process for the image region; and an em- 
bedding section, an'anged to embed the authentication 
information in the image region. 

[0028] An embodiment is operable to protect an 
image for each image region. 

[0029] The present invention provides an image 
processing apparatus comprising: an input section, ar- 
ranged to input digital image information; a detector, ar- 
ranged to recognize a predetermined image region in- 
cluded in the input image information; a generator, ar- 
ranged to generate authentication infomnation required 
to control a process for the image region; and an em- 



bedding section, arranged to embed the authentication 
information in the image region. In an embodiment, the 
generator and the embedding section generate and em- 
bed the authentication infomnation for each predeter- 

5 mined image region. 

[0030] Other features and advantages will be appar- 
ent from the following deschption taken in conjunction 
with the accompanying drawings, in which like reference 
characters designate the same or similar parts through- 

10 out the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 
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Fig. 1 shows an image processing system accord- 
ing to the first embodiment; 
Fig. 2 is a block diagram that expresses principal 
part of the arrangement shown in Fig. 1 as function 
modules; 

Fig. 3 is a flow chart showing the operation se- 
quence of the first embodiment; 
Fig, 4 is a flow chart showing the operation se- 
quence of the second embodiment; 
Figs. 5 and 6 show examples of mixed images; 
Fig. 7 is a block diagram showing the arrangement 
of an image processing system according to the 
third embodiment; 

Fig. 8 is a block diagram showing the arrangement 
of an MFP; 

Figs. 9 to 1 2 are flow charts for explaining an outline 
of the processes by the image processing system; 
Fig. 13 is a view for explaining block selection; 
Figs. 14A and 14B show the block selection result; 
Fig. 1 5 is a view for explaining embedding of a doc- 
ument watermark; 

Fig. 1 6 is a view for explaining extraction of a doc- 
ument watermark; 

Fig. 1 7 is a view for explaining the embedding rule 
of a document watermark; 

Fig. 1 8 is a flow chart showing the process for ex- 
ploring a shift amount; 

Fig. 1 9 is a block diagram showing the arrangement 

of an embedding processor (function module) for 

embedding watennark information; 

Fig. 20 is a block diagram showing details of a digital 

watermark generator; 

Fig. 21 shows examples of basic matrices; 

Fig. 22 shows an example of a digital watermark; 

Fig. 23 is a view showing an embedding process of 

a digital watermark; 

Fig. 24 shows an example of the configuration of 
image data; 

Fig. 25 is a block diagram showing the arrangement 
of an extraction processor (function module) for ex- 
tracting watermark information embedded In an im- 
age; 

Fig. 26 is a block diagram showing details of proc- 
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esses of an extraction pattern generator; 

Fig. 27 shows examples of extraction patterns; 

Fig. 28 is a view for explaining an integrated image; 

Fig. 29 shows an example of extraction of a digital 

watermark; 

Fig. 30 is a sectional view showing an example of 

the structure of a digital copying machine; 

Fig. 31 is a flow chart showing the process for hiding 

an area-designated image, which is executed by an 

image processor in a reader unit; 

Fig. 32 shows an outline of a console; 

Fig. 33 is a flow chart for explaining the processes 

in steps S105 to S111 shown in Fig. 31 in detail; 

Fig. 34 depicts the contents of the flow shown in Fig. 

33; 

Fig. 35 depicts the contents of the flow shown in Fig. 
33; 

Figs. 36 to 38 are views for explaining a method of 
converting encoded data to bitmap data: and 
Fig. 39 is a flow chart for explaining the method of 
reconstructing an original image from bitmap data, 
which is executed by the image processor in the 
reader unit. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0032] An image processing apparatus according to 
embodiments of the present invention will be described 
in detail hereinafter with reference to the accompanying 
drawings. 

First Embodiment 

[Arrangement] 

[0033] Fig. 1 shows an image processing system ac- 
cording to the first embodiment. 

[0034] A computer system (personal computer) 1 and 
an image input apparatus (color image scanner) 2 arc 
connected via a cable 3 used to exchange data between 
them. Furthermore, the personal computer 1 and an im- 
age output apparatus (color printer) 4 are connected via 
a cable 5 used to exchange data between them. 
[0035] Fig. 2 is a block diagram that expresses prin- 
cipal part of the arrangement shown in Fig. 1 as function 
modules. 

[0036] Referring to Fig. 2, a CPU 11 controls opera- 
tions of other building components via a bus 20 in ac- 
cordance with a program stored in a ROM 13 or hard 
disk device 1 8 using a RAM 1 2 as a work memory. Fur- 
themriore, the CPU 11 controls an I/O 22 for the image 
Input apparatus in accordance with an instruction input 
from a keyboard and mouse 16 connected to an I/O 17 
for an operation input device so as to make the scanner 
2 capture an image. The CPU 11 then controls a display 
controller 14 to display the captured Image on a display 
15. Or the CPU 11 controls an I/O 19 for an external 



storage device to store the captured image in the hard 
disk device 18. Or the CPU 11 outputs the captured im- 
age to various networks via an interface (l/F) 23. Of 
course, an image may be acquired from a server con- 

5 nected to the network via the l/F 23. 

[0037] As the l/Os for the scanner 2 and printer 4. and 
also those for the hard disk device 1 8 and operation in- 
put device, a USB (Universal Serial Bus), IEEE1394 se- 
rial bus, or the like are suitably used. Also, as the I/O for 

10 the printer 4, an IEEE1284 interface may be used. 

[Operation] 

[0038] Fig, 3 is a flow chart showing the operation se- 

15 quence of the first embodiment. A program which de- 
scribes the sequence shown in Fig. 3 and can be exe- 
cuted by a computer is pre-stored in the ROM 13 or hard 
disk device 18. The CPU 1 1 loads that program onto the 
RAM 12 and executes the loaded program, thus execut- 

20 ing operations to be described below. 

[0039] Upon reception of an image output instruction 
(SI 0), the operation shown in Fig. 3 starts. More specif- 
ically, when the operator has issued an instruction for 
printing an image created using image edit process soft- 

25 ware or the like by the printer 4 by operating the key- 
board and mouse 16, the CPU 11 loads image data to 
be processed from the hard disk device 1 8 or the like 
onto the RAM 12 in accordance with this instruction. 
[0040] Note that an instruction may instruct to print an 

30 image that has already been rendered on the RAM 12 
by Image edit process software or the like. Also, an in- 
struction may instruct to scan an image from a print by 
the scanner 2 and to print the scanned image. Further- 
more, an instruction may instruct to download an image 

35 from a server connected to the network, and to print the 
downloaded image. A detailed description of these proc- 
esses will be omitted. 

[0041] The CPU 11 identifies the features of images 
contained in image data loaded onto the RAM 12, di- 

40 vidcs the image data into image regions, i.e., a multi- 
valued image region (including a photo image or the 
like), and a binary image region (including text, line im- 
ages, figures/tables, and the like), and writes the divi- 
sion results in a predetermined area on the RAM 12 

45 (S20). 

[0042] The CPU 11 determines based on the image 
region division results and image data held in the RAM 
12 whether or not the image to be processed includes 
a multi-valued image region (S30). If YES in step S30, 
50 the flow advances to step S40; otherwise, the flow jumps 
to step S50. 

[0043] In step S40, the CPU 11 embeds a digital wa- 
termark indicating an object to be copyrighted or the like 
in each multi-valued image region using the method of 
55 embedding a digital watermark in the spatial domain or 
the method of embedding a digital watermark In the fre- 
quency domain. 

[0044] The CPU 1 1 then determines based on the im- 
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age region division results and image data held in the 
RAM 12 whether or not the image to be processed in- 
cludes a binary Image region (S50). If YES in step S50, 
the flow advances to step S60; otherwise, the flow jumps 
to step S70. 

[0045] In step S60, the CPU 11 embeds a digital wa- 
termark indicalirg an object to be copyrighted or the like 
in each binary image region using the method of manip- 
ulating the spaces between neighboring characters or 
the method of forming a binary image using binary cells 
(density pattern) each consisting of 2 x 2 pixels. 
[0046] The CPU 11 integrates partial images of the 
multi-valued and binary image regions embedded with 
the digital watermarks to generate image data corre- 
sponding to one page to be printed on the RAM 12 
(S70). The CPU 11 then executes processes required 
upon printing the generated image data by the printer 4 
(e.g., a halftone process such as an error diffusion proc- 
ess or the like, gamma correction, conversion into page 
description language data, and the like), and sends gen- 
erated print data to the printer 4, thus making the printer 
4 print an image (S80). 

[0047] Note that region division In step S20 can adopt 
methods disclosed In, e.g., Japanese Patent Laid-Open 
Nos. 8-186706 and 8-336040. Japanese Patent Laid- 
Open No. 8-1 86706 discloses a method of extracting Im- 
age regions from a digital color data on which image re- 
gions having different components (corresponding to 
partial image regions) are mixed paying special atten- 
tion to an image region serving as a background (base), 
and determining if each of these image regions is a color 
photo, a color character or line image other than black, 
a monochrome density (gray) photo, a monochrome 
character or line image, or the like. Also, Japanese Pat- 
ent Laid-Open No. 8-336040 discloses a method of sat- 
isfactorily dividing a digital color image Into Image re- 
gions (region division) at high speed irrespective of the 
Input image size on the basis of the technique described 
in Japanese Patent Laid-Open No. 8-186706. 
[0048] In the above embodiment, a digital watermark 
is embedded in each of partial images of an image which 
Is created In advance using image edit process software 
or the like. Upon editing each partial Image by image 
edit process software, a feature of that partial Image 
may be identified, and a digital watermark may be em- 
bedded In the partial Image by an embedding method 
corresponding to the identification result. In this case, 
processes corresponding to steps S30 to S70 shown in 
Fig. 3 may be programmed in Image edit process soft- 
ware. 

Second Embodiment 

[0049] An Image processing apparatus according to 
the second embodiment of the present invention will be 
described below. Note that the same reference numer- 
als in the second embodiment denote substantially the 
same building components as those in the first embod- 



iment, and a detailed description thereof will be omitted. 
[0050] Fig. 4 is a flow chart showing the operation se- 
quence of the second embodiment. 
[0051] Upon reception of an image output instruction 

5 (S110), the operation shown In Fig. 4 starts. More spe- 
cifically, when the operator has issued an instruction for 
editing an image scanned from a print or the like by op- 
erating the keyboard and mouse 16, the CPU 11 loads 
Image data to be processed from the hard disk device 

10 1 8 or the like onto the RAM 12 in accordance with this 
instruction. 

[0052] As in the first embodiment, an instruction may 
instruct to edit an Image that has already been rendered 
on the RAM 12 by image edit process software or the 

15 like. Also, an instruction may instruct to scan an image 
from a print by the scanner 2 and to print the scanned 
image. Furthermore, an instruction may instruct to 
download an image from a server connected to the net- 
work, and to print the downloaded image. A detailed de- 

20 scriptlon of these processes will be omitted. 

[0053] The CPU 11 identifies the features of images 
contained in image data loaded onto the RAM 12, di- 
vides the image data Into image regions, I.e., a multi- 
valued image region (including a photo image or the 

25 like), and a binary image region (including text, line im- 
ages, figures/tables, and the like), and writes the divi- 
sion results In a predetennined area on the RAM 12 
(S120). 

[0054] The CPU 11 determines based on the image 
30 region division results and Image data held in the RAM 
12 whether or not the image to be processed includes 
a multi-valued image region (S130). If YES in step SI 30, 
the flow advances to step S140; otherwise, the flow 
jumps to step S150. 
35 [0055] The CPU 11 checks in step SI 40 if a digital 
watennark is embedded In each multi-valued image re- 
gion of the Image data held in the RAM 12. If YES In 
step S140, the CPU 11 writes that digital watermark In 
a predetermined area on the RAM 12 for each image 
region; othcnA/isc, the CPU 1 1 writes data Indicating that 
no digital watennark is embedded, In the predetermined 
area. 

[0056] The CPU 1 1 then determines based on the im- 
age region division results and image data held In the 
45 RAM 12 whether or not the image to be processed in- 
cludes a binary image region (SI 50). If YES in step 
S150, the flow advances to step SI 60; othenwise, the 
flow jumps to step SI 70. 

[0057] The CPU checks In step SI 60 if a digital wa- 
50 termark is embedded In each binary image region of the 
image data held in the RAM 12. If YES in step S1 60, the 
CPU 1 1 writes that digital watermark In a predetermined 
area on the RAM 12 for each Image region; otherwise, 
the CPU 11 writes data indicating that no digital water- 
55 mark is embedded, in the predetermined area. 

[0058] The CPU 11 Integrates the extracted digital 
watennarks (8170) and checks if the Image designated 
by the print instruction is an object to be copyrighted or 
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the like (SI 80). If it is determined that the image itself 
or its partial image is to be copyrighted or the like, the 
flow advances to step S200; otherwise, the flow advanc- 
es to step SI 90. 

[0059] In step S190, the CPU 11 allows the user to 
edil Ihe image. In this manner, the operalorcan use (clip, 
edil, prim, save, and the like) Ihe desired image. On Ihe 
other hand, in step S200 the CPU 11 displays a mes- 
sage indicating that the image designated by the print 
instruction or its partial image is an object to be copy- 
righted or the like on tho display 1 5 to issue an alert to 
the operator, thus ending tho process. 
[0060] In the above embodiment, digital watermarks 
are extracted from respective partial images of an image 
scanned by the scanner 2. Alternatively, upon editing 
each partial image by image edit process software, a 
feature of that partial image may be identified, and a dig- 
ital watermark which is embedded by an embedding 
method corresponding to the identification result may be 
extracted from the partial image. In this case, processes 
corresponding to steps S130 to S200 shown in Fig. 4 
may be programmed in image edit process software. 
[0061] Furthermore, a digital watermark may be 
checked for each image region in place of integrating 
the extracted digital watemnarks, and the availability of 
each image region (partial image) may be determined. 
Use of a partial image includes a clipping process of a 
partial image, an edit process of the clipped partial im- 
age, a print or save process of the clipped image, and 
the like. 

Third Embodiment 
[Arrangement] 

[0062] Fig. 7 is a block diagram showing the arrange- 
ment of an image processing system according to the 
third embodiment. 

[0063] This image processing system is implemented 
in an environment in which offices (a plurality of sections 
like offices) 130 and 120 are connected via a WAN 1 04 
(e.g., the Internet). 

[0064] To a LAN 1 07 formed in the office 1 30, an MFP 
(Multi-Function Processor) 100, a management PC 101 
for controlling the MFP 1 00, a client PC 1 02, a document 
management server 106. a database 105 to be man- 
aged by the document management server, and the like 
are connected. The office 120 has substantially the 
same arrangement as that of the office 1 30, and at least 
a document management server 1 06, a database 1 05 
to be managed by the document management server, 
and the like are connected to a LAN 108 fomned in the 
office 120. The LANs 107 and 1 08 of the offices 130 and 
120 are interconnected via a proxy server 103 connect- 
ed to the LAN 107, the WAN 104, and a proxy server 
103 connected to the LAN 108. 

[0065] The MFP 100 has charge of some of image 
processes for scanning an image on a paper document, 



and processing the scanned image An image signal 
output from the MFP 100 is input to the management 
PC 101 via a communication line 109. The management 
PC 101 comprises a general personal computer (PC), 

5 which has a memory such as a hard disk or the like for 
storing an image, an image processor implemented by 
hardware or software, a monitor such as a CRT, LCD, 
or the like, and an input unit including a mouse, key- 
board, and the like, and some of these units are inte- 

10 grated to the MFP 1 00. 

[0066] Fig. 8 is a block diagram showing the arrange- 
ment of the MFP 100. 

[0067] An image reading unit 110 including an auto 
document feeder (ADF) irradiates an image on each of 

15 one or a plurality of stacked documents with light coming 
from a light source, forms an image of light reflected by 
the document on a solid-state image sensing element 
via a lens, and obtains a scanned image signal (e.g., 
600 dpi) in the raster order from the solid-state Image 

20 sensing element. Upon copying a document, the 
scanned image signal is converted into a recording sig- 
nal by a data processing unit 115. Upon copying a doc- 
ument onto a plurality of recording sheets, a recording 
signal for one page is temporarily stored in a storage 

25 unit 111 , and is repetitively output to a recording unit 112, 
thus forming images on the plurality of recording sheets. 
[0068] On the other hand, print data output from the 
client PC 102 Is input to a network interface (l/F) 114 via 
the LAN 107, and is converted Into recordable raster da- 

30 ta by the data processing unit 115. The raster data Is 
formed as an image on a recording sheet by the record- 
ing unit 112. 

[0069] The operator Inputs an Instruction to the MFP 

1 00 using a key console equipped on the MFP 1 00 and 
35 an input unit 113 including a keyboard and mouse of the 

management PC 1 01 . A display unit 11 6 displays an op- 
eration input and image process status. 
[0070] The operation of the MFP 1 00 is controlled by 
a controller (not shown) In the data processing unit 115. 
40 [0071 ] Note that tho storage unit 1 1 1 can also be con- 
trolled from the management PC 101. Data exchange 
and control between the M FP 1 00 and management PC 

1 01 are done via a network l/F 1 1 7 and a signal line 1 09 
that directly couples them. 

45 

[Process] 

[0072] Figs. 9 to 12 are flow charts for explaining an 
outline of the processes by the aforementioned image 
so processing system. 

[0073] The image reading unit 1 1 0 scans a document 
to obtain a 600-dpi, 8-bit image signal (Image informa- 
tion input process, 81 201 ). The data processing unit 1 1 5 
executes pre-processes such as trimming, skew correc- 
ts tion (including correction of a direction), noise removal, 
and the like for the image signal (S1202), generates a 
binary Image by a binarization process (SI 203), and 
saves Image data (multi-valued and binary Image data) 
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for one page in the storage unit 111 . 
[0074] A CPU of the managennent PC 1 01 executes 
block selection for the image data stored in the storage 
unit 1 1 1 to identify a text/line image part, halftone image 
part, and background part where neither a text/line im- 
age nor an image are present (SI 204). Furthermore, the 
CPU divides the text/line image part into regions (text 
regions) for respective paragraphs and other structures 
(a table or line image having ruled lines). On the other 
hand, the CPU divides the halftone image part and back- 
ground part into independent objects (picture regions) 
for respective minimum division units such as rectangu- 
lar regions and the like (SI 205). Then, the CPU extracts 
a binary image for each text region and a multi-valued 
image for each picture region from the image data 
stored in the storage unit 1 1 1 on the basis of the position 
information of the divided(detected) text and picture re- 
gions (S1 206). I n the following description, the extracted 
image region will also be referred to as a "block". 
[0075] The following processes are done for each 
block. Whether or not watermark information is embed- 
ded in the block to be processed is determined by a doc- 
ument watermark detection process when the block to 
be processed is a text region or a background water- 
mark detection process when it is a picture region 
(S1207). If it is determined that watermark information 
is embedded, a display flag of the region of interest is 
set to OFF (S1210); othenwise, a display flag of the re- 
gion of interest is set to ON (S1209). It Is then checked 
if the same process has been done for all blocks 
(S1211). The processes in steps S1207 to 81210 are 
repeated until the display flags of all the blocks are set. 
[0076] Subsequently, a block to be processed is se- 
lected (S1212), and it is checked based on the display 
flag if watermark information is embedded in the select- 
ed block (SI 213). If no watermark information is embed- 
ded, the flow jumps to "process A" to be described later. 
On the other hand, if watermark infonnation is embed- 
ded, the control prompts the operator to input a pass- 
word (SI 21 4). This password is used to control display 
of the block of interest, and to authenticate other control 
functions such as print, send, and the like, as will be 
described later. 

[0077] If a password is input its authenticity is 
checked (S1215). If a wrong password is input, the flow 
advances to "process B" to be described later. If a cor- 
rect password is input, it is checked if that password is 
a display password (S1216). If YES in step SI 21 6, it is 
further checked if the block of interest corresponds to 
the background part (SI 21 7). If the block of interest cor- 
responds to a part (text region or halftone image part) 
other than the background part, the display flag of that 
block is set to ON (S1221). 

[0078] If it is determined In step SI 21 7 that the block 
of interest corresponds to the background part, i.e., the 
background part embedded with watermark informa- 
tion, since that part includes no image, pointer informa- 
tion indicating the storage location of original data of an 



image is extracted from the watermark information (to 
be referred to as a "background watermark" hereinafter) 
embedded in the background (SI 21 8), and the original 
data is acquired from the document management server 

5 106 or the like (S1219). In this case, if no watermark 
information is embedded in original data, succession of 
the watermark information is required. If the original data 
does not succeed any watermark information, various 
kinds of control of the block of interest are disabled. Al- 

10 ternatively, new watemriark information may be input in 
place of succession. The original data of the block of 
interest succeeds the watermark information (e.g., 
background watermark infomnation is embedded in an 
image as an invisible watermark) or new watermark in- 

15 formation is embedded in the original data (S1220) to 
prepare for a display image embedded with the water- 
mark information. After that, the display flag of the block 
of interest is set to ON (SI 221 ). 

[0079] On the other hand, if it is determined in step 
20 SI 21 6 that the password Is not a display password, It is 
checked if the block of interest is a text region (SI 222). 
If NO in step S1222, the flow jumps to step S1225. If 
YES in step S1222, binary image data of the block of 
interest is sent to and saved in the document manage- 
rs ment server 106 or the like (SI 223), and watermark In- 
formation (including pointer information indicating the 
storage location of image data, various passwords, var- 
ious kinds of control information, and the like) is embed- 
ded as a background watermark to mask the block of 
30 interest (SI 224). In step S1225, the display flag of the 
block of interest is set to ON. 

[0080] Other kinds of control information (availability 
of charge, print, copy, send, and the like) of the block of 
interest are extracted from the watermark information 

35 (S1226), and other control flags are set ON or OFF in 
accordance with the control information (SI 227). It is 
then determined whether or not the processes of all 
blocks are complete (SI 228). If NO In step S1228, the 
flow returns to step SI 21 2; otherwise, various kinds of 

'^o controls arc made in accordance with the control flags 
(1229). Note that a chargeflag,printflag, copy flag, send 
flag, and the like corresponding to control information of 
charge, print, copy, send, and the like are prepared. If 
these flags are ON, image data of the block of interest 

45 Is printed, copied, or transmitted; if these flags are OFF, 
image data of the block of Interest is not printed, copied, 
or transmitted. 

[0081] "Process A" to be executed when it is deter- 
mined in step SI 21 3 that no watermark information is 

50 embedded will be explained below. 

[0082] It is checked if the block of interest is a text 
region (SI 241). If NO in step SI 241 , since the block of 
interest is not the one to be controlled, the flow advances 
to step SI 228. On the other hand, if YES in step SI 241 , 

55 a watermark embedding mode is set to prompt the user 
to select whether a document watermark that allows to 
read text is embedded (display mode) or a background 
watemnark is embedded to mask the block of interest 
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(non-display mode) (SI 242). If the user has selected the 
display nnode, various passwords are set (SI 246), and 
waternnark infomnation containing these passwords is 
embedded as a document watermark (S1247). On the 
other hand, if the user has selected the non-display 
mode, various passwords are set (S1243), and binary 
image data of the block of interest is sent to and saved 
in the document management server 106 or the like 
(SI 244). Then, a background watermark that contains 
pointer information , various passwords, various kinds of 
control information^ and the like is embedded to mask 
the block of interest (SI 245). 

[0083] The image (the image after the waternnark in- 
formation is embedded or background) of the block of 
interest is re-displayed (SI 248), and the flow advances 
to step S1228 

[0084] "Process B" to be executed when it is deter- 
mined in step SI 21 5 that a wrong password is input will 
be described below. 

[0085] It is checked if the block of interest is a text 
region (S1251 ). If the block of interest is not a text region 
(security about display is maintained since that region 
is originally masked), all control flags are set to OFF to 
maintain security about control (SI 255), and the flow 
advances to step SI 228. If the block of interest is a text 
region, binary image data of the block of interest is sent 
to and saved in the document management server 1 06 
or the like (SI 252) so as not to display that block. A 
background watermark containing pointer information of 
the storage location, various passwords, various kinds 
of control information, and the like is embedded to mask 
the block of interest (SI 253), and the block of interest 
Is re-displayed (SI 254). Then, all control flags are set 
to OFF (S1255), and the flow advances to step S1228. 
[0086] As an example of various kinds of control, print 
and send limitations will be explained below. 

• Upon receiving a print instruction, 

a block with an OFF print flag: a background 
image embedded with a background water- 
mark Is printed; or 

a block with an ON print flag: an image embed- 
ded with a document watermark or an image of 
original data is printed. 

• Upon receiving a send instruction, 

a block with an OFF send flag: a background 
image embedded with a background water- 
mark is transmitted; or 

a block with an ON send flag: an image embed- 
ded with a document watermark or original data 
is transmitted. 

[0087] With this control, security can be freely man- 
aged (e.g., to impose a browse limitation, copy limita- 
tion, send limitation, print limitation, to charge for reuse 



and the like) for each object of a document. Upon print- 
ing a document, since a document watermark and invis- 
ible watermark are respectively embedded in a text re- 
gion and picture region, security management of objects 
5 scanned from a printed image can be Implemented, thus 
greatly improving Ihe document security. 
[0088] Principal processes will be described in more 
detail below. 

10 [Block Selection] 

[0089] Block selection in steps S1 204 and S1 205 will 
be explained first. 

[0090] Block selection is a process for recognizing an 

15 image for one page shown in Fig. 1 3 as a set of objects, 
determining the property of each object to TEXT, PIC- 
TURE, PHOTO, LINE, orTABLE, and dividing the image 
into regions (blocks) having different properties. An ex- 
ample of block selection will be explained below. 

20 [0091] An image to be processed is binarized to a 
monochrome image, and a cluster of pixels bounded by 
black pixels is extracted by contour tracing. For a cluster 
of black pixels with a large area, contour tracing is made 
for white pixels in the cluster to extract clusters of white 

25 pixels. Furthermore, a cluster of black pixels in the clus- 
ter of white pixels with a predetermined area or more is 
extracted. In this way, extraction of clusters of black and 
white pixels are recursively repeated. 
[0092] The obtained pixel clusters are classified Into 

30 regions having different properties in accordance with 
their sizes and shapes. For example, a pixel cluster 
which has an aspect ratio close to 1 , and has a size that 
falls within a predetemnlned range is determined as that 
of a text property. Furthermore, when neighboring pixel 

35 clusters with a lexl property line up and can be grouped, 
they are determined as a text region. Also, a low-profile 
pixel cluster with a small aspect ratio is categorized as 
a line region, a range occupied by black pixel clusters 
that include white pixel clusters which have a shape 

40 close to a rectangle and lino up is categorized as a table 
region, a region where pixel clusters with indeterminate 
forms are distributed is categorized as a photo region, 
and other pixel clusters with an arbitrary shape is cate- 
gorized as a picture region. 

45 [0093] Figs. 14A and 1 48 show the block selection re- 
sults. Fig. 14A shows block information of each of ex- 
tracted blocks. Fig. 14B shows input file information and 
indicates the total number of blocks extracted by block 
selection. These pieces of information are used upon 

50 embedding or extracting watermark information. 

[Embedding Process of Document Watermark] 

[0094] The embedding process of a document water- 
55 mark will be explained below. 

[0095] A document image 3001 shown in Fig. 15 is a 
block separated as a text region by block selection. Fur- 
thermore, circumscribing rectangles 3004 for respective 
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text elements are extracted from the text region by a 
document image analysis process 3002. A text element 
Indicates a rectangular region extracted using projec- 
tion, and corresponds to either one character or a build- 
ing component (radical or the like) of a character 5 
[0096] The space lengths between neighboring cir- 
cumscribing rectangles are calculated on the basis of 
information of the extracted circumscribing rectangles 
3004, and respective circumscribing rectangles are 
shifted to the right or left on the basis of embedding rules io 
to be described later to embed 1 -bit information between 
neighboring circumscribing rectangles (embedding 
process 3003), thereby generating a document image 
3005 embedded with watermark infomiation 3006. 
[0097] The document image analysis process 3002 is 15 
an element technique of character recognition, and is a 
technique for dividing a document Image into a text re- 
gion, a figure region such as a graph or the like, and the 
like, and extracting respective characters one by one in 
the text region using projection. For example, a tech- 20 
nique described in Japanese Patent Laid-Open No. 
6-68301 may be adopted. 

[Extraction Process of Document Watennark] 

25 

[0098] The extraction method of a document water- 
mark will be explained below. 

[0099] As in the document watermark embedding 
process, circumschbing rectangles 3103 of characters 
are extracted from an image 3005 shown in Fig. 16 by 30 
block selection and a document image analysis process 
3002, and the space lengths between neighboring cir- 
cumscribing rectangles are calculated on the basis of 
infomiation of the extracted circumscribing rectangles 
3103. In each line, a character used to embed 1-bit in- 35 
formation is specified, and embedded watermark infor- 
mation 31 05 is extracted on the basis of the embedding 
rules to be described later (extraction process 31 04). 
[0100] The embedding rules will be explained below. 
[0101] Let P and S be the space lengths before and 40 
after a character where 1-bit information is embedded, 
as shown in Fig. 17. One-bit infonnation is embedded 
in every other characters except for the two end char- 
acters of a line. (P - S)/(P + S) is calculated from the 
space lengths, and the result is quanti7ed by an appro- ^5 
priate quantization step to calculate the remainder thus 
reconstructing 1-bit information. Equation (1) repre- 
sents this relationship, and can extract an embedded 
value V('0' or'1'). 

50 

V = floor[(P - S)/{a(P + S)}] mod2 (1 ) 

where a is the quantization step (0 < a < 1 ), 
[0102] Upon embedding watennark information, a cir- 55 
cumscribing rectangle is shifted to the right or left pixel 
by pixel, and a shift amount (the number of pixels) to the 
left or right is increased until a value ('0' or '1 ') is obtained 



by equation (1). 

[0103] Fig. 1 8 is a flow chart showing the process for 
exploring the shift amount. In Fig. 18, variable i indicates 
a candidate value of the shift amount, and variables 
Flag1 and Flag2 indicate whether or not a character to 
be shifted touches a neighboring character if it is shifted 
distance i to the right or left. If the character to be shifted 
touches a neighboring character, variable Flagi or 
Flag2 assumes 'V. 

[0104] Initial values of variables are set (S3402), and 
it is determined whether or not a character (or character 
clement) to be shifted touches a right neighboring char- 
acter (or character element) if it is shifted distance i to 
the right (S3403). If YES in step S3403, Flagi is set to 
'1' (S3404). Subsequently, it is determined whether or 
not the characterto be shifted touches a left neighboring 
character if it is shifted distance i to the left (S3405). If 
YES in step S3405, Flag2 is set to 'V (83406). 
[0105] It is then checked if it Is possible to shift the 
character distance i (S3407). If both the flags are '1', it 
is determined that it is impossible to shift the character, 
and the shift amount is set to zero (S3408). In this case, 
it is impossible to embed infonnation by shifting the 
character to be shifted. 

[0106] If Flagi = '0' (S3409), whether or not value V 
to be embedded is obtained by shifting the character to 
be shifted distance i to the right is detennined using 
equation (1) (S3410). If YES in step S3410, the shift 
amount is set to +i (S3411 ). Note that a positive sign of 
the shift amount indicates right shift, and a negative sign 
indicates left shift. 

[0107] If Flagi = '1', or if value V cannot be obtained 
by right shift and Flag2 = '0' (S341 2), whether or not val- 
ue V to be embedded is obtained by shifting the char- 
acter to be shifted distance i to the left Is determined 
using equation (1) (S3413). If YES in step 83413, the 
shift amount is set to -i (83414). 

[01 08] If neither right shift nor left shift can yield value 
V, variable i is incremented (S341 5), and the flow returns 
to step S3403. 

[0109] A character is shifted in accordance with the 
shift amount explored in this way, thus embedding 1 -bit 
information. By repeating the aforementioned process 
for respective characters, watermark information is em- 
bedded in a document image. 

[Digital Watermark Embedding Processor] 

[0110] A digital watermark to be described below Is 
also called an "invisible digital watennark", and is a 
change itself in original image data as small as a person 
can hardly perceive. One or a combination of such 
changes represent arbitrary additional information. 
[0111] Fig. 19 is a block diagram showing the ar- 
rangement of an embedding processor (function mod- 
ule) for embedding watennark Infonnation. 
[0112] The embedding processor comprises an im- 
age input unit 4001, embedding information input unit 
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4002, key information input unit 4003, digital watermark 
generation unit 4004, digital watermark embedding unit 
4005, and image output unit 4006. Note that the digital 
watermark embedding process may be implemented by 
software with the above arrangement. 
[0113] The image inpul unil 4001 inputs image data I 
of an image in which watermark information is lo be em- 
bedded. In the following description, assume that image 
data I represents a monochrome multi-valued image for 
the sake of simplicity. Of course, when watermark infor- 
mation Is embedded in imago data such as color Image 
data consisting of a plurality of color components, each 
of R, G, and B components or luminance and color dif- 
ference components as the plurality of color compo- 
nents is handled In the same manner as a monochrome 
multi-valued image, and watermark Infomnation can be 
embedded In each component. In this case, watermark 
infomiatlon with an Information size three times that of 
a monochrome multi-valued image can be embedded. 
[0114] The embedding infomnation input unit 4002 In- 
puts watermark Information to be embedded in image 
data I as a binary data sequence. This binary data se- 
quence will be referred to as additional Information Inf 
hereinafter. Additional information Inf is formed of a 
combination of bits each of which Indicates either '0' or 
•1'. Additional information Inf represents authentication 
information used to control a region corresponding to 
image data I, pointer Information to original data, or the 
like. A case will be exemplified below wherein additional 
infonnation Inf expressed by n bits is to be embedded. 
[0115] Note that additional information Inf may be en- 
crypted not to be readily misused. Also, additional infor- 
mation Inf may undergo error correction coding so as to 
correctly extract additional information Inf even when 
image data I has been changed (lo be referred to as 
"attack" hereinafter) so as not to extract additional infor- 
mation I from it. Note that some attacks may be not de- 
liberate. For example, watermark information may be re- 
moved as a result of general image processes such as 
Irreversible compression, luminance correction, geo- 
metric transformation, filtering, and the like. Since proc- 
esses such as encryption, error correction coding, and 
the like are known to those who are skilled in the art, a 
detailed description thereof will be omitted. 
[0116] The key Information input unit 4003 inputs key 
information k required to embed and extract additional 
infomnation Inf. Key infonnation k is expressed by L bits, 
and is, e.g., "01 01 01 01 " ("85" in decimal notation) if L = 
8. Key information k is given as an initial value of a pseu- 
do random number generation process executed by a 
pseudo random number generator 4102 (to be de- 
scribed later). As long as the embedding processor and 
an extraction processor (to be described later) use com- 
mon key infomnation k, embedded additional informa- 
tion Inf can be correctly extracted. In other words, only 
a user who possesses key infonnation k can correctly 
extract additional infomnation Inf. 

[0117] The digital watermark generation unit 4004 re- 



ceives additional information Inf from the embedding in- 
formation input unit 4002, and key information k from 
the key information input unit 4003, and generates dig- 
ital watennark w on the basis of additional information 
5 Inf and key infonnation k. Fig. 20 Is a block diagram 
showing details of the digital watermark generation unil 
4004. 

[0118] A basic matrix generator 41 01 generates basic 
matrix m. Basic matrix m is used to specify correspond- 

10 ence between the positions of bits which form additional 
Information Inf, and the pixel positions of image data I 
where respective bits arc to be embedded. The basic 
matrix generator 4101 can selectively use a plurality of 
basic matrices, and a basic matrix to be used must be 

15 changed in correspondence with the purpose intended/ 
situation. By switching a basic matrix, optimal water- 
mark information (additional information Inf) can be em- 
bedded. 

[0119] Fig. 21 shows examples of basic matrices m. 
20 A matrix 4201 is an example of basic matrix m used up- 
on embedding 16-bit additional information Inf, and nu- 
merals ranging from 1 to 1 6 are assigned to 4 x 4 ele- 
ments. The values of elements of basic matrix m corre- 
spond to the bit positions of additional information Inf. 
25 That is, bit position "1 " (most significant bit) of additional 
information Inf corresponds to a position where the val- 
ue of an element of basic matrix m Is "1" and, likewise, 
bit position "2" (bit next to the most significant bit) of ad- 
ditional information Inf corresponds to a position where 
30 the value of an element of basic matrix m is "2". 

[0120] A matrix 4202 is an example of basic matrix m 
used upon embedding 8-bit additional infomnation Inf. 
According to the matrix 4202, 8 bits of additional infor- 
mation Inf correspond to elements having values rang- 
es ing from "1" lo "8" of Ihose of the matrix 4201, and no 
bit positions of additional information Inf correspond to 
elements which do not have any value. As shown in the 
matrix 4202, by scattering positions corresponding to re- 
spective bits of additional information Inf, a change In 
40 imago (image quality deterioration) upon embedding 
additional infomnation Inf can be harder to recognize 
than the matrix 4201 . 

[0121] A matrix 4203 is another example of basic ma- 
trix m used upon embedding 8-bit additional information 

45 Inf as In the matrix 4202 According to the matrix 4202, 
1 -bit information is embedded in one pixel. However, ac- 
cording to the matrix 4203. 1-bit information is embed- 
ded in two pixels. In other words, the matnx 4202 uses 
50% of all pixels to embed additional infomnation Inf, 

50 while the matrix 4203 uses all pixels (100%) to embed 
additional information Inf. Hence, when the matrix 4203 
is used, the number of times of embedding additional 
information Inf is increased, and additional information 
Inf can be extracted more reliably (higher robustness 

55 against attacks is obtained) than the matrices 4201 and 
4202. Note that the ratio of pixels used to embed water- 
mark information will be referred to as a "filling ratio" 
hereinafter Note that the filling ratio of the matrix 4201 
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is 1 00%, that of the matrix 4202 is 50%, and that of the 
matrix 4203 is 100%. 

[01 22] A matrix 4202 can embed only 4-bit additional 
infomnation Inf although it has a filling ratio of 100%. 
Hence, 1 -bit information is embedded using four pixels, 5 
and the number of limes of embedding additional infor- 
mation Inf is further increased to further improve the ro- 
bustness against attacks, but the information size that 
can be embedded becomes smaller than other matri- 
ces. 10 
[0123] In this manner by selecting the configuration 
of basic matrix m, the filling ratio, the number of pixels 
to be used to embed 1 bit, and the information size that 
can be embedded can be selectively set. The filling ratio 
influences the image quality of an image in which wa- 15 
termark information is embedded, and the number of 
pixels used to embed 1 bit mainly influences the robust- 
ness against attacks. Therefore, the image quality de- 
terioration is emphasized with increasing filling ratio. Al- 
so, the robustness against attacks becomes higher and 20 
the information size that can be embedded decreases 
with increasing number of pixels used to embed 1 bit. In 
this manner, the Image quality, robustness against at- 
tacks, and information size have a trade-off relationship. 
[0124] In the third embodiment, the robustness 25 
against attacks, image quality, and information size can 
be controlled and set by adaptlvety selecting a plurality 
of types of basic matnces m. 

[0125] The pseudo random number generator 41 02 
generates pseudo random number sequence r on the 30 
basis of input key information k. Pseudo random number 
sequence r is a real number sequence according to a 
uniform distribution included within the range {-1,1), and 
key infomnation k is used as an initial value upon gen- 
erating pseudo random number sequence r. That is, 35 
pseudo random number sequence r(k1) generated us- 
ing key information k1 is different from pseudo random 
number sequence r(k2) generated using key informa- 
tion k2 k1 ). Since a method of generating pseudo ran- 
dom number sequence r is known to those who arc 
skilled in the art, a detailed deschption thereof will be 
omitted. 

[01 26] A pseudo random number assignment section 
4103 receives watermark information Inf, basic matrix 
m , and pseudo random number sequence r, and assigns 45 
respective bits of watermark infomnation Inf to respec- 
tive elements of pseudo random number sequence r on 
the basis of basic matrix m, thus generating digital wa- 
termark w. More specifically, respective elements of a 
matrix 4204 are scanned in the raster order to assign 50 
the most significant bit to an element having value "1 
the second most significant bit to an element having val- 
ue "2", and so forth. If a given bit of additional information 
Inf is '1', the corresponding element of pseudo random 
number sequence r is left unchanged; if it is '0', the cor- 55 
responding element of pseudo random number se- 
quence r is multiplied by -1 . By repeating the aforemen- 
tioned process for n bits of additional information I nf , dig- 
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ital watermark w exemplified in Fig. 22 is obtained. Note 
that digital watermark w shown in Fig. 22 is generated 
when, for example, basic matrix m is the matrix 4204 
shown in Fig. 21 , the pseudo random numbersequence 
is r= {0.7, -0.6, -0.9, 0.8}, and additional Information Inf 
(4 bits) is "1001". 

[0127] In the above example, 4 x 4 basic matrices m 
are used to embed additional information Inf each con- 
sisting of 1 6 bits, 8 bits, and 4 bits. However the present 
invention is not limited to such specific example. For ex- 
ample, more pixels may bo used to 1-bit information, 
and basic matrix m with a larger size may be used. If 
basic matrix m with a larger size is used, pseudo random 
number sequence r uses a longer real number se- 
quence. In practice, the aforementioned random 
number sequence which consists of four elements may 
disturb a normal function of an extraction process (to be 
described later). That is. although additional information 
Inf is embedded, correlation coefficients between inte- 
grated image c and digital watermarks w1 , w2 wn 

may become small. Hence, in order to embed, e.g., 
64-blt additional Information, 256 x 256 basic matrix m 
is used at a filling ratio of 50%. In this case, 512 pixels 
are used to embed 1 bit. 

[0128] The digital watennark embedding unit 4005 re- 
ceives image data I and digital watemiark w, and outputs 
image data I* embedded with digital watermark w The 
digital watermark embedding unit 4005 executes a dig- 
ital watermark embedding process according to: 



where I'j j is the image data embedded with the digital 
watermark, 

li j Is the Image data before the digital watermark 
is embedded, 

Wj j is the digital watermark, 

i and j are x- and y-coordinate values of the image 
or digital watermark, and 

a is a parameter for setting the strength of the dig- 
ital watermark. 

[0129] As parameter a, for example, a value around 
"10" may be selected. By increasing a, a digital water- 
mark with higher robustness against attacks can be em- 
bedded, but Image quality deterioration becomes larger. 
On the other hand, by decreasing a, the robustness 
against attacks is decreased, but image quality deterio- 
ration can be suppressed. As in the configuration of ba- 
sic matrix m, the balance between the robustness 
against attacks and image quality can be adjusted by 
appropriately setting the value a. 
[0130] Fig. 23 shows the digital watemnark embed- 
ding process given by equation (2) above in detail. Ref- 
erence numeral 4401 denotes image data I' embedded 
with the digital watermark; 4402, image data I before the 
digital watemnark is embedded; and 4403, digital water- 
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mark w. As shown in Fig. 23, arithnnetic operations of 
equation (2) are made for respective elements in the 
matrix. 

[0131] The process given by equation (2) and shown 
in Fig. 23 is repeated for whole image data I. If image 
data I is made up of 24 / 24 pixels, as shown in Fig. 24, 
il is broken up into blocks (macroblocks) of 4 a 4 pixels, 
which do not overlap each other, and the process given 
by equation (2) is executed for each macroblock. 
[0132] By repeating the digital watermark embedding 
process for all macrobiocks, watermark information can 
be consequently embedded in the entire imago. Since 
one macroblock is embedded with additional informa- 
tion Inf consisting of n bits, embedded additional infor- 
mation Inf can be extracted if there is at least one mac- 
roblock. In other words, the extraction process of addi- 
tional information Inf does not require the entire image, 
and only a portion of Image data (at least one macrob- 
lock) suffices to execute that process. A feature that ad- 
ditional infonnation Inf can be extracted from a portion 
of image data I will be referred to as "having clipping 
robustness" hereinafter. 

[0133] Generated image data 1' embedded with addi- 
tional information Inf as a digital watermark becomes a 
final output of the embedding processor via the image 
output unit 4006. 

[Digital Watermark Extraction Processor] 

[0134] Fig. 25 is a block diagram showing the ar- 
rangement of the extraction processor (function mod- 
ule) for extracting watermark infonnnation embedded in 
an image. 

[0135] The embedding processor comprises an im- 
age input unit 4601 , key information input unit 4602, dig- 
ital watermark (extraction pattem) generation unit 4603, 
digital watermark extraction unit 4604, and digital wa- 
termark output unit 4605. Note that the digital watermark 
extraction process may be implemented by software 
having the aforementioned arrangement. 
[01 36] The image input unit 4601 receives image data 
1" in which watermark information may be embedded. 
Note that image data 1" input to the image input unit 4601 
may be any of image data 1' embedded with watermark 
information by the aforementioned embedding proces- 
sor, attacked image data 1', and image data I in which 
no watermark information is embedded. 
[0137] The key information input unit 4602 receives 
key infonnation k required to extract watermark informa- 
tion. Note that key information k input to this unit must 
be the same one input to the key information input unit 
4003 of the aforementioned embedding processor. If dif- 
ferent key information is input, additional information 
cannot be nomnally extracted. In other words, only the 
user who has correct key information k can extract cor- 
rect additional information Inf. 

[0138] The extraction pattern generation unit 4603 re- 
ceives key information k, and generates an extraction 



pattern on the basis of key information k. Fig. 26 shows 
details of the process of the extraction pattern genera- 
tion unit 4603. The extraction pattern generation unit 
4603 comprises a basic matrix generator 4701 , pseudo 

5 random number generator 4702, and pseudo random 
number assignment section 4703. Since the basic ma- 
trix generator 4701 and pseudo random number gener- 
ator 4702 execute the same operations as those of the 
aforementioned basic matrix generator 41 01 and pseu- 

^0 do random number generator 41 02, a detailed descrip- 
tion thereof will be omitted. Note that additional informa- 
tion cannot be normally extracted unless the basic ma- 
trix generators 4701 and 4101 generate identical basic 
matrix m on the basis of identical key information k. 

^5 [0139] The pseudo random number assignment sec- 
tion 4703 receives basic matrix m and pseudo random 
number sequence r, and assigns respective elements 
of pseudo random number sequence r to predetermined 
elements of basic matrix m. The difference between this 

20 assignment section 4703 and the pseudo random 
number assignment section 41 03 in the aforementioned 
embedding processor is that the pseudo random 
number assignment section 4103 outputs only one dig- 
ital watennark w, while the pseudo random number as- 

25 signment section 4703 outputs extraction patterns wn 
corresponding to the number of bits of additional infor- 
mation Inf (for n bits in this case). 
[0140] Details of a process for assigning respective 
elements of pseudo random number sequence r to pre- 

50 detemnined elements of basic matrix m will be explained 
taking the matrix 4204 shown in Fig. 1 5 as an example. 
When the matrix 4204 is used, since 4-bit additional in- 
formation Inf can be embedded, four extraction patterns 
w1 , w2, w3, and w4 are output. More specifically, re- 

35 speclive elements of the matrix 4204 are scanned in the 
raster order to assign respective elements of pseudo 
random number sequence r to those having a value "1 ". 
Upon completion of assignment of respective elements 
of pseudo random number sequence r to all elements 

40 having a value "1", a matrix to which pseudo random 
number sequence r is assigned is generated as extrac- 
tion pattern w1. Fig. 27 shows examples of extraction 
patterns, and shows a case wherein a real number se- 
quence r = {0.7, -0.6, -0.9, 0.8} is used as pseudo ran- 

45 dom number sequence r. The aforementioned process 
is repeated for elements having values "2", "3", and "4" 
of the matrix 4204 to generate extraction patterns w2, 
w3, and w4, respectively. By superposing extraction pat- 
terns w1 , w2, w3, and w4 generated in this way, a pat- 

50 tern equal to digital watemnark w generated by the em- 
bedding processor is obtained. 

[0141] The digital watermark extraction unit 4604 re- 
ceives image data 1" and extraction patterns w1 , w2:..., 
wn, and extracts additional information Inf from image 
55 data I". In this case, it is desired that additional informa- 
tion Inf to be extracted is equal to embedded additional 
information Inf. However, if image data 1' has suffered 
various attacks, these pieces of information do not al- 
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ways match. 

[01 42] The digital watermark extraction unit 4604 cal- 
culates correlation values between integrated image c 
generated from image data 1" and extraction patterns 
w1 , w2,..., wn. Integrated imagec is obtained by dividing 5 
image data I" into macroblocks, and calculating Ihe av- 
erage of element values of each macroblock. Fig. 28 is 
a view for explaining integrated imagec when extraction 
patterns of 4 x 4 pixels, and image data I" of 24 > 24 
pixels are input. Image data I" shown in Fig. 28 is broken io 
up into 36 macroblocks, and integrated imago c is ob- 
tained by calculating the average values of respective 
element values of these 36 macroblocks. 
[0143] Correlation values between integrated image 
c generated in this way, and extraction patterns w1, 15 
\n2,. ., wn are calculated respectively. A correlation co- 
efficient is a statistical quantity used to measure similar- 
ity between integrated image c and extraction pattern 
wn, and Is given by: 

20 

p = c''^.w'n/lc'^llw'nl (3) 

where c' and w'n are matrices each of which has as el- 
ements the differences between respective elements 
and the average values of the elements, and 

c'^ Is the transposed matrix of c'. 
[0144] Correlation coefficient p assumes a value 
ranging from -1 to +1 . If positive correlation between in- 
tegrated image c and extraction pattern wn is strong, p 
approaches +1 ; If negative correlation is strong, p ap- 
proaches -1 . "Positive correlation is strong" means that 
"extraction pattern wn becomes larger with increasing 
integrated image c", and "negative correlation is strong" 
means that "extraction pattern wn becomes smaller with 
increasing integrated image c". When integrated image 
c and extraction pattern wn have no correlation, p = 0. 
[0145] Based on the correlation values calculated in 
this way, whether or not additional information Inf is em- 
bedded in image data I" and whether each bit that fonns 
additional information Inf is '1 ' or '0' If the additional in- 
formation is embedded are determined. That is, the cor- 
relation coefficients between integrated image c and ex- 
traction patterns w1 , w2 .., wn arc calculated, and if 
each calculated correlation coefficient is close to zero, 
it is determined that "no additional infomnation is em- 
bedded"; if each correlation coefficient is a positive 
number separated from zero, it is determined that '1' is 
embedded; and if each correlation coefficient is a neg> 
ative number separated from zero, it is determined that 
'0' is embedded. 

[0146] Calculating correlation is equivalent to evalua- 
tion of similarities between integrated image c and ex- 
traction patterns w1 , w2,..., wn. That is, when the afore- 
mentioned embedding processor embeds a pattern cor- 

responding to extraction patterns w1 , w2 wn in image 

data 1" (integrated image c), correlation values indicat- 



ing higher similarities are calculated. 
[01 47] Fig. 29 shows an example wherein a digital wa- 
termark is extracted from image data I" (integrated im- 
age c) embedded with 4-bit additional infomriation using 
w1 , w2, w3, and w4. 

[0148] The correlation values between integrated im- 
age c and four extraction patterns w1 , w2,..., wn are re- 
spectively calculated. When additional information Inf 
is embedded in image data 1" (integrated image c), cor- 
relation values are calculated as, e.g., 0.9, -0.8, -0.85, 
and 0.7. Based on this calculation result, it Is determined 
that additional information Inf is "1001", and 4-bit addi- 
tional infonnation Inf can be finally output. 
[0149] Extracted n-bit additional information Inf is 
output as an extraction result of the extraction processor 
via the digital watermark output unit 4605. In this case, 
if the embedding processor has made an error correc- 
tion encoding process and encryption process upon em- 
bedding additional information Inf, an error correction 
decoding process and decryption process are executed. 
The obtained infonnation is finally output as a binary da- 
ta sequence (additional information Inf). 

Modification of Third Embodiment 

[0150] In the above description, a document water- 
mark and background watemriark are selectively used 
as watermarks. However, the present invention is not 
limited to such specific watermarks, and watermarking 
schemes optimal to respective objects may be selec- 
tively used. 

[01 51 ] Also, authentication control is implemented us- 
ing a password. However, the present invention is not 
limited to such specific example, but authentication con- 
trol may be implemented by key control. 

Fourth Embodiment 

[0152] An image processing apparatus according to 
the fourth embodiment of the present invention will bo 
described below. Note that the same reference numer- 
als in the fourth embodiment denote substantially the 
same building components as those in the third embod- 
iment, and a detailed description thereof will be omitted. 

[Arrangement] 

[0153] Fig. 30 shows the outer appearance of the 
structure of a digital copying machine. The digital cop- 
ying machine comprises a reader unit 51 which digitally 
scans a document image, and generates digital image 
data by way of a predetermined image process, and a 
printer unit 52 which generates a copy image based on 
the generated digital image data. 
[0154] A document feeder 51 01 of the reader unit 51 
feeds documents one by one in turn from the last page 
onto a platen glass 5102. Upon completion of reading 
of each document image, the feeder 5101 exhausts a 
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document on the platen glass 5102. When a docunnent 
is fed onto the platen glass 51 02, a lamp 51 03 is turned 
on, and a scanner unit 5104 begins to move, thus ex- 
posing and scanning the document. Light reflected by 
the document at this time is guided to a CCD image sen- 
sor (lo be referred to as "CCD" hereinafter) 51 09 via mir- 
rors 5105, 5106, and 5107, and a lens 5108 so as to 
form an optical image on it. In this way, the scanned doc- 
ument image is read by the CCD 5109, and an image 
signal output from the CCD 5109 undergoes image 
processes such as shading correction, sharpness cor- 
rection, and the like by an image processor 5110. After 
that, the processed image signal is transferred to the 
printer unit 52. 

[0155] A laser driver 5221 of the printer unit 52 drives 
a laser emission unit 5201 in accordance with image da- 
ta input from the reader unit 51 . A laser beam output 
from the laser emission unit 5201 scans a photosensi- 
tive drum 5202 via a polygonal mirror thereby forming 
a latent image on the photosensitive drum 5202. The 
latent image formed on the photosensitive drum 5202 is 
applied with a developing agent (toner) by a developer 
5203 to form a toner image. 

[0156] A recording sheet fed from a cassette 5204 or 
5205 is conveyed to a transfer unit 5206 in synchronism 
with the beginning of irradiation of the laser beam, and 
the toner image applied to the photosensitive drum 5202 
is transferred onto the recording sheet. The recording 
sheet on which the toner image has been transferred is 
conveyed to a fixing unit 5207, and the toner image is 
fixed to the recording sheet by heat and pressure of the 
fixing unit 5207. The recording sheet which has left the 
fixing unit 5207 is exhausted by exhaust rollers 5208. A 
sorter 5220 sorts recording sheets by storing exhausted 
recording sheets on respective bins. Note that the sorter 
5220 stores recording sheets on the uppermost bin if a 
sort mode is not selected. 

[0157] If a both-side recording mode is set, after the 
recording sheet is conveyed to the position of the ex- 
haust rollers 5208, it is guided onto a rc-fccd paper con- 
vey path by the exhaust rollers 5208 which are rotated 
in the reverse direction, and a flapper 5209. If a multiple 
recording mode is set, the recording sheet is guided on- 
to the re-feed paper convey path before it is conveyed 
to the exhaust rollers 5208. The recording sheet con- 
veyed onto the re-feed paper convey path is fed to the 
transfer unit 5206 at the aforementioned timirrg. 

fProcessl 

[0158] Fig. 31 is a flow chart showing the process for 

hiding an area-designated image, which is executed by 
the image processor 5110 in the reader unit 51 . 
[0159] Upon reception of an image signal from a doc- 
ument, the image processor 5110 generates digital im- 
age data obtained by normally quantizing luminance in- 
formation for respective fine pixels at a precision of 
about 8 bits (S101). The spatial resolution of a pixel is 



around 42 |xm > 42 |.im, and corresponds to a resolution 
of about 600 pixels per inch (25.4 mm) (600 dpi). The 
image processor 5110 displays an image represented 
by the generated image data on a screen of a console 

5 shown in Fig. 32. 

[0160] The console normally comprises a liquid crys- 
tal display, the surface of which is covered by a touch 
panel, and allows the user to make desired operations 
by operating buttons displayed on the screen. Referring 

10 to Fig. 32, buttons 601 are used to select an apparatus 
mode: a "copy" mode copies a road document image 
(by outputting it from the printer unit 52), a "transmit" 
mode transmits image data of the read image to a re- 
mote place via a network as a digital file, and a "save" 

15 mode saves image data of the read image in an auxiliary 
storage device such as a hard disk or the like incorpo- 
rated in the apparatus as a digital file. In this case, as- 
sume that the "copy" mode has been selected, and its 
button frame is indicated by a bold line. 

20 [0161] A display unit 602 displays basic operation 
conditions of the apparatus in accordance with the se- 
lected mode. Upon selection of the copy mode, the dis- 
play unit 602 displays the output recording sheet size 
and enlargement/reduction scale. A preview display unit 

25 603 displays the entire image read by the reader unit 51 
in a reduced scale. A frame 604 displayed on the pre- 
view display unit 603 indicates an area set on a preview- 
displayed image. The size of an area indicated by the 
frame 604 (to be simply referred to as "area" hereinafter) 

30 is determined by operating one of buttons 605, and the 
area moves vertically or horizontally upon operation of 
each of buttons 606. In other words, the size and posi- 
tion of the area 604 on the preview display unit 603 
change upon operation of the buttons 605 and 606. 

35 [0162] A box 607 is used lo input authentication infor- 
mation (to be described later). For example, a character 
string of four digits Is input using a ten-key pad (not 
shown), and symbols or the like corresponding in 
number to the digits of the input character string are dis- 

40 played. The reason why symbols "*" arc displayed in 
place of directly displaying the input character string is 
to improve security. 

[0163] The image processor 5110 accepts the area 
604 and authentication information which are designat- 
es edand input by the user using the console (S1 02. SI 03). 
Upon completion of designation and input, the image 
processor 5110 extracts image data designated by the 
area 604 from the input image data (SI 04), detemnines 
the type of extracted image data (SI 05), selects an im- 
50 age compression method corresponding to the determi- 
nation result (8106), and compresses the extracted im- 
age data by the selected image compression method 
(SI 07). Then, the image processor 5110 generates 
code data by synthesizing an identification code that in- 
55 dicates the image compression method used, and the 
input authentication information (SI 08), and converts 
the generated code data into bitmap data by a method 
to be described later (SI 09). The image processor 511 0 
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erases image data in the area 604 from the input image 
data (S11 1 ), synthesizes image data embedded with the 
bitmap code data obtained in step S1 09 to a blank area 
after erasure (S11 2), and outputs the synthesized image 
data (S113). 

[0164] In Ihis case, since the copy mode is selected 
as the apparatus mode, Ihe output image data is sent 
to the printer unit 52, and a copy Image is formed on a 
recording sheet. Likewise, if the transmit mode is select- 
ed as the apparatus mode, the output image data is sent 
to a network communication unit, and is digitally trans- 
ferred to a predetermined destination. If the save mode 
is selected, the output image data Is stored in an auxil- 
iary storage device in the apparatus. 
[0165] Fig. 33 is a flow chart for explaining the proc- 
esses in steps S105 to S112 in detail. 
[0166] Initially, the type of extracted image data Is de- 
termined. In this case, it is checked if the area-designat- 
ed image is a continuous tone image such as a photo 
or the like or a binary image such as a text/line image 
(S203). As a detemiination method, various methods 
such as a method using a histogram Indicating the lu- 
minance distribution of an objective image, a method us- 
ing the frequencies of occurrence for respective spatial 
frequency components, a method using whether or not 
an objective image is more likely to be recognized as 
"line" by pattern matching, and the like have been pro- 
posed, and such known methods can be used. 
[0167] If it is determined that the extracted image is a 
text/line image, a histogram indicating the luminance 
distribution of the image Is generated (S204), and an 
optimal threshold value that can be used to separate the 
background and text/line image is calculated based on 
this histogram (S205). Using this threshold value, the 
image data is binarized (S206), and the obtained binary 
image data undergoes a compression process (S207). 
This compression process can adopt a known binary im- 
age compression method. Normally, as a binary image 
compression method, one of lossless compression 
methods froofrom any losses of information (e.g., MM R 
compression. MR compression, MH compression, JBIG 
compression, and the like) is adaptlvely used. Of 
course, it is possible to adaptively use one of the above 
methods so as to minimize the code size after compres- 
sion. 

[0168] On the other hand, if it Is determined that the 
extracted image is a continuous tone image, resolution 
conversion is made (S208). The input image data is read 
at. e.g., 600 dpi. However, it is usually the case that a 
halftone image such as a photo or the like does not ap- 
pear to deteriorate at about 300 dpi. Hence, in order to 
reduce the final code size, the image data is converted 
into that corresponding to 300 dpi by reducing the ver- 
tical and horizontal sizes to 1/2. The 300-dpl multi-val- 
ued image data then undergoes a compression process 
(S209). As a compression method suited to a multi-val- 
ued image, known JPEG compression, J PEG2000 com- 
pression, and the like can be used. Note that these com- 



pression methods are lossy ones in which an original 
image suffers deterioration which is normally visually 
imperceptible. 

[0169] Code information used to identify the compres- 
5 sion method is appended to the obtained compressed 
image data (S210). This information is required lo des- 
ignate an expansion method upon reconstructing an 
original image from an output Image. For example, the 
following identification codes are assigned In advance 
10 to the respective compression methods: 

JPEG compression BB 

15 

JPEG2000 CO 



M MR compression DO 



MH compression — > EE 



JBIG compression — > FF 

[0170] Then, a code of authentication information is 
appended (S211). The authentication information is re- 
quired to discriminate if a person who is about to recon- 
struct an image has the authority of doing it upon recon- 
structing an original Image from an output image. Only 
when authentication Infonnatlon appended In this step 
Is correctly designated upon reconstruction, a recon- 
struction process to an original image is executed. 
[0171] A digital signal sequence of the code data ob- 
tained in this way is converted as a binary number Into 
binary bitmap data (S21 2), and is synthesized to the ar- 
ea 604 by embedding (S213). 

[0172] Fig. 34 depicts the aforementioned operations, 
when the area 604 is designated on input image data 
301 , image data within the area 604 is erased, and Is 
replaced by bitmap code data 

[0173] Fig. 35 depicts the contents of the flow shown 
in Fig. 33. 

[0174] An image of Sx x Sy pixels in the area 604 is 
extracted, and since it is determined that this extracted 
image is a text/iine image, the image undergoes binari- 
zation and lossless compression. An identification code 
of the compression method is appended to, e.g., the 
head of the compressed code sequence, and authenti- 
cation Information Is also appended to the head of the 
resultant code sequence. After binarization and bitmap 
conversion, bitmap data having the same size as the 
area 604, I.e., Sx x Sy pixels, is generated, and replac- 
es an Image In the area 604. Of course, the appending 
positions of the Identification code and authentication 
information are not limited to the head of the code se- 
quence. For example, the identification code and au- 
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thentication information may be appended to other ar- 
bitrarily predetermined positions (e.g., the end of the 
code sequence or predetermined bit positions). Further- 
more, the identification code and authentication infor- 
mation may be repetitively appended to a plurality of po- 
sitions to make sure extraction of them. 

[Bitmap Conversion of Code Data] 

[0175] Figs. 36 to 38 are views for explaining the 
method of converting code data into bitmap data, and 
show three different methods. In each of Figs. 36 to 38, 
a small rectangle indicates one pixel at 600 dpi. 
[0176] in the method shown in Fig. 36, pixels at 600 
dpi are converted into bitmap data so that 2 >1 pixels 
have 1 -bit information If code data (left side) expressed 
as a binary number is '1', four (2 x 2) pixels are set to 
'1' (black); if code data Is '0', four pixels are set to '0' 
(white). Consequently, binary bitmap data having a res- 
olution (300 dpi) 1 12 of 600 dpi is generated. The reason 
why 2x2 pixels are used to express 1-bit information 
is to eliminate the influences of the reading precision, 
positional deviation, magnification error, and the like of 
the reader and to accurately reconstruct code data from 
a bitmap image upon reconstructing an original image 
by scanning a bitmap image printed on a recording 
sheet by the reader according to this embodiment. 
[0177] In the method shown in Fig. 37, in place of set- 
ting all of 2 X 2 pixels to have identical values, if code 
data is '1V the upper left small pixel (corresponding to 
600 dpi) of four pixels Is set to '1 ' (black); if code data is 
'OV the lower right small pixel is set to '1' (black). With 
this configuration, the reliability upon reconstructing an 
original image by scanning the printed bitmap image can 
be improved. 

[0178] In the method shown In Fig. 38, 1 bit Is ex- 
pressed by 4 X 2 pixels, and '1 ' and '0' are expressed by 
layouts of black and white pixels shown in Fig. 38, With 
this configuration, the data size that can be recorded per 
unit area is reduced, but the reading precision upon re- 
constructing an original image can be further improved. 
[0179] Note that the bitmap conversion method is not 
limited to those described above, and various other 
methods may be used. 

[0180] The si7e of bitmap data to be generated, and 
the size of Infomnation that can be embedded in that data 

will be described below. 

[0181] Assuming that the area 604 has a size of 2" x 
2" (about 5 cm in both the vertical and horizontal direc- 
tions) on a document, since original image data is 600 
dpi, each of Sx and Sy amounts to 1200 pixels. That is, 
if 8 bits are assigned per pixel, the infomnation size of 
image data in the area 604 is: 

1200 X 1200 X 8 - 1 1 ,520,000 bits = 1 1 M bits 

[0182] When code data is converted into bitmap data 



by one of the aforementioned methods, and replaces an 
image in the area 604, since the methods of Figs. 36 
and 37 embed 1-bit information using four pixels, the 
size of information that can be recorded is reduced to 
5 1/4x1/8 = 1/32, and the data size that can be embed- 
ded in the 2" x 2" area 604 is: 

11M/32= 0.34M bits 

10 

[0183] Put differently, it is impractical since image da- 
ta of 11 M bits must bo compressed to 1/32, i.e., 0.34M 
bits. For this reason, the image property of the area 604 
must be determined to adaptively switch the binariza- 

^5 tion, resolution conversion, and compression method. If 
an image in the area 604 is a text/line image, it is bina- 
rized while the resolution of 600 dpi is kept unchanged. 
As a result, the data size of the image can be reduced 
to 1/8, i.e., (11/8 =) 1 .38M bits. In order to further reduce 

20 this data size to 0.34M bits, 1/4 compression is required. 
However, this compression ratio can be easily achieved 
by MMR or JBIG compression. Of course, since the 
code information of the compression method, authenti- 
cation infonnation, and the like must also be embedded, 

25 a compression ratio higher than 1/4 is required, but such 
ratio can still relatively easily be achieved. 
[0184] On the other hand, in case of a photo/halftone 
Image, the resolution is halved (300 dpi) while the 
number of gray scales of 8 bits remains unchanged, 

30 thereby reducing the data size to 1/4, i.e., (11/4=)2.75M 
bits. In order to further reduce this data size to 0.34M 
bits, 1/8 compression is required. However, this com- 
pression ratio can be achieved by JPEG or JPEG2000 
very easily while suppressing image quality deteriora- 

35 lion. 

[0185] If the bitmap conversion method shown in Fig. 
38 is adopted, the size of Information that can be em- 
bedded is further reduced to 1/2, and the compression 
ratio must be doubled. However, this bitmap conversion 
40 method docs not yield an impractical value as the afore- 
mentioned compression method. 

[Reconstruction of Original Image] 

45 [0186] Fig. 39 is a flow chart for explaining the method 
of reconstructing an original image from bitmap data, 
which is executed by the image processor 5110 in the 
reader unit 51 . 

[0187] The image processor 5110 inputs an image 
50 (S801 ). If an image on a printout is to be input, that im- 
age can be read by the reader unit 51 and can be input 
as a digital image; if an image is digitally transmitted or 
saved, it can be directly input as a digital image. 
[0188] The image processor 5110 detects a hidden 
55 image area from the input image (S802). This detection 
adopts a method of, e.g., detecting a rectangular region 
included in the input image, and determining the hidden 
image area if periodic patterns of black and white pixels 
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are present in the detected rectangular region. 
[0189] The image processor 5110 reads a pixel se- 
quence fronn Image data of the detected, hidden Image 
area (S803), and determines a bitmap conversion meth- 
od of that image data to reconstruct a binary code se- 
quence (S804). From the code sequence, the image 
processorSII 0 extracts an idenlificallon code indicating 
a compression method (S805), and also authentication 
information (S806). 

[0190] Next, the image processor 5110 displays a 
message the input image includes a hidden image on, 
e.g., the screen of the console, and prompts the user to 
input authentication information required to reconstruct 
an Image (S807). If the user inputs the authentication 
information, the image processor 511 0 checks if the in- 
put authentication information matches the extracted 
authentication information (S808). If they do not match, 
the image processor 51 1 0 directly outputs the input Im- 
age (S813). 

[0191] If the two pieces of authentication information 
match, the image processor 5110 reconstructs an orig- 
inal Image. In this case, the image processor 5110 ex- 
tracts code data of a compressed image except for the 
identification code of the compression method and au- 
thentication information from the code sequence 
(S809), and applies an expansion process of the conri- 
presslon method corresponding to the extracted identi- 
fication code to the extracted code data (S81 0). The im- 
age processor 51 1 0 then replaces the image of the de- 
tected, hidden image area by the expanded image 

(5811) , and outputs the obtained synthesized image 

(581 2) . On the image to be output in this step, an original 
Image before the area-designated Image Is hidden Is re- 
constructed. 

[0192] In this way, by converting code data obtained 
by efficiently compressing an area-designated partial 
image into bitmap data, and synthesizing the bitmap da- 
ta to an original image, the area-designated image can 
be hidden by replacing it by a visually unidentifiable im- 
ago. If such unidentifiable imago (hidden image area) is 
found, the image of that area is recognized (decoded) 
as code data, and an original image can be reconstruct- 
ed by a user who has the authority of browsing or the 
like on the basis of the identification code of the com- 
pression method set in the code data with reference to 
authentication infonnatlon set in that code data. 
[0193] Hence, the user who has the predetermined 
authority can reconstruct an original image, and can dis- 
play, print, copy, transmit, and/or save the ohginal Im- 
age. Note that authentication infonnation may be inde- 
pendently set for each of Image operations, i.e., charge, 
display, print, copy, send, and save operations, or may 
be set together for each of groups of image operations 
such as display and print, copy and send, and the like. 

Modification of Fourth Embodiment 

[0194] In the above description, one area 64 is desig- 



nated to hide an image of that area, as shown in Fig. 34 
and the like. However, the number of areas to be hidden 
is not limited to one, and a plurality of areas can be des- 
ignated. In this case, the processes in steps SI 02 to 

5 S1 1 2 can be repeated for respective designated areas. 
When an original image is reconstructed from an image 
having a plurality of hidden image areas, the processes 
in steps S803 to S811 can be repeated for respective 
detected, hidden image areas. 

'0 [0195] In the above description, the information hiding 
method, encoding method, and reconstruction method 
for a document imago to bo road by the digital copying 
machine have been explained. However, these meth- 
ods can also be applied to documents, figures, and the 

15 like on a PC (personal computer). In this case, when the 
user instructs to print a document or figure, a device driv- 
er corresponding to a printer which is used to print is 
launched, and generates image data for a printout on 
the basis of print code generated by an application on 

20 the PC. The device driver displays the generated image 
data for preview on its user interface window, as shown 
in Fig. 32, and accepts designation of the area 604 that 
the user wants to hide, and input of authentication infor- 
mation. Or the device driver detects the hidden image 

25 area. The subsequent processes are the same those 
described above, but are Implemented by the device 
driver on the PC (more specifically, a CPU which exe- 
cutes device driver software). 

[0196] In the above description, code data is convert- 
30 ed into bitmap data. However, an original image cannot 
often be accurately reconstructed due to distortion of a 
printed Image, stains on a recording sheet, and the like. 
In order to avoid such troubles. If code data is converted 
into bitmap data after an error correction code Is ap- 
35 pended to the code data, the reliability of data recorded 
as a bitmap can be improved. Since various known 
methods have been proposed for error correction 
codes, such methods can be used. In this case, howev- 
er, since the size of information that can be enlbedded 
40 is reduced, a higher compression ratio of an Imago must 
be set accordingly. Of course, in addition to the error 
correction code, code data may be converted into bit- 
map data after it is encrypted, so as to improve robust- 
ness against information leakage. 

45 

<Other Embodiment> 

[0197] Embodiments of the invention may comprise a 
system constituted by a plurality of devices (e.g., host 
50 computer, interface, reader, printer) or an apparatus 
comprising a single device (e.g., copying machine, fac- 
simile machine). 

[0198] Further, embodiments can also be achieved by 
providing a carrier (storage medium or signal) carrying 
55 program codes for performing the aforesaid processes 
to a computer system or apparatus (e.g., a personal 
computer), reading the program codes, by a CPU or 
MPU of the computer system or apparatus, from the car- 
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rier, then executing the progrann. 
[0199] In this case, the program codes read from the 
carrier realize the functions according to the embodi- 
ments, and the carrier carrying the program codes 
therefore constitutes an embodiment. 
[0200] A storage medium, such as a floppy disk, a 
hard disk, an optica! disk, a magneto-optical disk, 
CD-ROM, CD-R, a magnetic tape, a non-volatile type 
memory card, and ROM can be used for providing the 
program codes. 

[0201] Furthermore, besides the aforesaid cases 
w/hcrc functions according to the above embodiments 
are realized by executing the program codes which are 
read by a computer, embodiments include a case where 
an OS (operating system) or the like working on the 
computer performs a part or entire processes in accord- 
ance with designations of the program codes and real- 
izes functions according to the above embodiments. 
[0202] Furthermore, embodiments Include a case 
where, after the program codes read from the storage 
medium are written in a function expansion card which 
is inserted into the computer or in a memory provided 
in a function expansion unit which is connected to the 
computer, CPU or the like contained in the function ex- 
pansion card or unit performs a part or entire process in 
accordance with designations of the program codes and 
realizes functions of the above embodiments. 
[0203] In a case where computer program code Is 
supplied on a carrier, the carrier stores program codes 
for programming a programmable processing appara- 
tus to become operable to perform according to the flow- 
charts described in the embodiments. 
[0204] The present invention is not limited to the 
above-described embodiments, and various changes 
and modifications can be made wilhin the scope of the 
claims. 



Claims 

1. An Image processing apparatus comprising: 

an input section, arranged to input digital image 
information; 

a detector, arranged to recognize a predeter- 
mined Image region Included In the input image 
information; 

a generator, arranged to generate authentica- 
tion information required to control a process 
for the image region; and 
an embedding section, arranged to embed the 
authentication information In the Image region. 

2. The apparatus according to claim 1, wherein said 
generator and said embedding section generate 
and embed the authentication Information for each 
predetermined Image region. 



3. A computer program product comprising a carrier 
carrying computer program code for an image 
processing method comprising process procedure 
codes for: 

5 

inputting digital image information; 
recognizing a predetermined image region in- 
cluded in the input image information; 
generating authentication information required 
10 to control a process for the image region; and 

embedding the authentication Information In 
the image region. 

4. An image processing method comprising the steps 
15 of: 

inputting digital image information; 
receiving information that designates a prede- 
termined region contained in the input image 
20 information; 

compressing image information In the desig- 
nated region; 

converting the compressed image information 
into code data which is to replace the image in- 
25 formation in the designated region; and 

replacing the image information in the designat- 
ed region by the code data. 

5. The method according to claim 4, further compris- 
30 ing the step of detennining a type of image informa- 
tion in the designated region, adaptively selecting 
a compression method on the basis of the deter- 
mined type, and appending information Indicating 
the selected compression method to the code data. 

35 

6. The method according to claim 4, further compris- 
ing the step of receiving Input of authentication in- 
formation required to reconstruct image information 
before compression from the code data, and ap- 

40 pending the input authentication information to the 

code data. 

7. The method according to claim 4, wherein the step 
of generating the code data includes a step of de- 

45 termining a type of image information in the desig- 
nated region, and adaptively executing binarization 
or resolution conversion on the basis of the deter- 
mined type, so as to reduce a data size of the code 
data to be generated to a value not more than a pre- 

50 determined value. 

8. The method according to claim 4, wherein the code 
data is image data in a bitmap format. 

55 9. An image processing method comprising the steps 
of: 

Inputting digital image information; 
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detecting a hidden image region included in the 

input image infomiation; 

extracting code data from the detected image 

region; 

reconstructing an Image by expanding the ex- 
tracted code data; and 

synlhesUing the reconstructed image with Ihe 
detected image region. 

10. The method according to claim 9, further compris- 
ing the stops of: 

extracting authentication information from the 
code data; 

receiving input of authentication infonnation; 
and 

executing expansion when the extracted au- 
thentication infonnation matches the input au- 
thentication information. 

11. An image processing apparatus comprising: 

an Input section, arranged to input digital Image 
Information; 

a reception section, arranged to receive infor- 
mation that designates a predetermined region 
contained in the input image information; 
a compressor, arranged to compress image in- 
formation in the designated region; 
a converter, arranged to convert the com- 
pressed image infonnation into code data 
which is to replace the Image Information in the 
designated region; and 

a replacement section, arranged to replace the 
image information in the designated region by 
the code data. 

12. An image processing apparatus comprising: 

an input section, arranged to input digital image 
information; 

a detector, arranged to detect a hidden image 
region included in the Input image infonnation; 
an extractor, arranged to extract code data from 
the detected image region; 
an expander, arranged to reconstruct an Image 
by expanding the extracted code data: and 
a synthesizer, arranged to synthesize the re- 
constructed Image with the detected image re- 
gion. 

13. A computer program product comprising a carrier 
carrying computer program code for an image 
processing method comprising process procedure 
codes for: 

inputting digital image information; 

receiving information that designates a prede- 



termined region contained in the input image 
infonnation; 

compressing image Information in the desig- 
nated region: 

5 converting the compressed image information 

into code data which is to replace Ihe image in- 
formation in the designated region; and 
replacing the Image information In the designat- 
ed region by the code data. 

10 

14. A computer program product comprising a carrier 
carrying computer program code for an image 
processing method comprising process procedure 
codes for: 

15 

inputting digital image information; 
detecting a hidden image region included in the 
input image information; 
extracting code data from the detected image 
20 region; 

reconstructing an Image by expanding the ex- 
tracted code data; and 

synthesizing the reconstructed image with the 
detected image region. 

25 

15. An image processing apparatus comprising: 

a detector, arranged to divide an input image 
into a plurality of image regions having different 

30 features; 

an embedding section, arranged to embed dig- 
ital watermarks in the respective detected Im- 
age regions by embedding methods according 
to the features of the image regions; and 

^5 an integrator arranged to integrate the image 

regions embedded with the digital watermarks 
Into one image. 

16. The apparatus according to claim 15, further com- 
'^o prising an output section, arranged to output data 

which makes an image output device form an image 
on the basis of the integrated image. 

17. The apparatus according to claim 15, wherein the 
45 image region includes a multi-valued image region 

and binary image region, and said detector divides 
the Image region into the multi-valued image region 
and binary image region. 

so 18. An image processing apparatus comprising: 

a detector, arranged to divide an input image 
into a plurality of image regions having different 
features; 

55 an extractor, arranged to extract digital water- 

marks embedded In the respective detected im- 
age regions by embedding methods according 
to the features of the Image region; and 



55 



20 



1349370A2 I > 



39 



EP 1 349 370 A2 



40 



a determiner, arranged to determine availability 
of the input image on the basis of the extracted 
digital watermarks. 

19. The apparatus according to claim 18, wherein said 

determiner determines availability of an image 
process for each of the detected image regions. 

20. The apparatus according to claim 18, further com- 
prising a controller arranged to control an image in- 
put device used to acquire an image from a print. 

21. The apparatus according to claim 18, wherein the 

image region includes a multi-valued image region 
and binary image region, and said detector divides 
the image region into the multi-valued image region 
and binary image region. 

22. An Image processing method comprising the steps 
of: 

dividing an input image into a plurality of image 
regions having different features; 
embedding digital watermarks in the respective 
detected Image regions by embedding meth- 
ods according to the features of the image re- 
gions; and 

integrating the image regions embedded with 
the digital watermarks into one image. 

23. An image processing method comprising the steps 
of: 

dividing an input image into a plurality of image 
regions having different features; 
extracting digital watermarks embedded in the 
respective detected image regions by embed- 
ding methods according to the features of the 
image region; and 

determining availability of the input Imago on 
the basis of the extracted digital watemnarks. 

24. The method according to claim 23, wherein the de- 

temnining step includes a step of determining avail- 
ability of an image process for each of the detected 
image regions. 

25. A computer program product comprising a carrier 
carrying computer program code for an image 
processing method comprising process procedure 
codes for: 

dividing an input image into a plurality of image 
regions having different features; 
embedding digital watermarks In the respective 
detected image regions by embedding meth- 
ods according to the features of the Image re- 
gions; and 



Integrating the image regions embedded with 
the digital watennarks Into one image. 

26. A computer program product comprising a carrier 
5 carrying computer program code for an image 

processing method comprising process procedure 
codes for: 

dividing an input image into a plurality of image 
10 regions having different features; 

extracting digital watermarks embedded In the 
respective detected image regions by embed- 
ding methods according to the features of the 
image region; and 
15 detemnining availability of the input image on 

the basis of the extracted digital watermarks. 

27. An image processing method comprising the steps 
of: 

20 

inputting digital image information; 
recognizing a predetermined Image region In- 
cluded In the input image information; 
generating authentication information required 
25 to control a process for the Image region; and 

embedding the authentication information in 
the image region; 

28. The method according to claim 27, wherein the au- 
30 thenticatlon infomnatlon is generated and embed- 
ded for each predetermined image region. 

29. The method according to claim 27, wherein the em- 
bedding step Includes a step of embedding the au- 

35 thenticatlon information as a digital watermark. 

30. The method according to claim 27, further compris- 
ing the steps of: 

40 extracting information embedded in the prede- 

termined Image region; and 
controlling the process for the image region on 
the basis of authentication information con- 
tained in the extracted information. 

45 

31. The method according to claim 27, wherein the 
process for the Image region includes at least one 
of a charge, display, print, copy, and send process- 
es. 

50 

32. A computer program product according to any of 
claims 3, 13, 14, 25 and 26 when embodied as a 
storage medium storing the code. 

55 33. A computer program product according to any of 
claims 3, 13, 14, 25 and 26 when embodied as a 
signal carrying the code. 
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